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ABSTRACT 

We present the results from a search of data from the first 33.5 days of the Kepler 
science mission (Quarter 1) for exoplanet transits by the Planet Hunters citizen science 
project. Planet Hunters enlists members of the general public to visually identify tran- 
sits in the publicly released Kepler light curves via the World Wide Web. Over 24,000 
volunteers reviewed the Kepler Quarter 1 data set. We examine the abundance of > 
2 R ffi planets on short period (< 15 days) orbits based on Planet Hunters detections. 
We present these results along with an analysis of the detection efficiency of human 
classifiers to identify planetary transits including a comparison to the Kepler inventory 
of planet candidates. Although performance drops rapidly for smaller radii, > 4 R ffi 
Planet Hunters > 85% efficient at identifying transit signals for planets with periods 
less than 15 days for the Kepler sample of target stars. Our high efficiency rate for 
simulated transits along with recovery of the majority of Kepler >4 R^ planets suggest 
suggests the Kepler inventory of >4 R ffi short period planets is nearly complete. 

Subject headings: Planets and satellites: detection-Planets and satellites: general 



1. Introduction 

In the past nearly two decades, there has been an explosion in the number of known planets 
orbiting stars beyond our own solar system, with over 700 extrasolar planets (exoplanets) known 
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to date ([Schneider! l201ll ; Wright et al .11 20 111) . While most early discoveries were made using the 
radial velocity method (jLovis fc Fischerl 120101 and references therein), the advent of large digital 
CCD-cameras has enabled astronomers to make effective use of the transit method, recording the 
dimming in light that is seen i f a planet p asses in front of its parent star as viewed from Earth. 
Ground-based transit surveys (jWinnl I201Q and references therein) have successfully detected the 
~1% drop in flux due to Jupiter and Neptune-size d planets, bu t we have ent ered a new era of 



exoplanet discovery with the spaced-based CoRoT (jBaglinl 120031 ) and Kepler (jKoch et al 



2010 



Borucki et al.ll2010bl ) telescopes surveying tens to hundreds of thousands of main sequence stars 
with photometric precision suf ficient to observe th e 0.0084% transit depth of an Earth-sized planet 
transiting a Sun-like star (e.g. iFressin et al.ll2012l ). 



Kepler is monitoring nearly continuously the brightness of more than 150,000 stars selected 
from a 115 deg 2 field in the constellations Cygnus, Lyra, and Draco with a 29.4 minute cadence . 
An automated rout i ne, th e Transiting Planet Search (TPS) algorithm (jTenenbaum et al. I boid : 
Jenkins et al.ll2010bl . |2002j) searches the data for periodic signals which might indicate the presence 
of a planet. In order to avoid being swamped with false positives, TPS requires a phase- folded 
detection statistical confidence better than 7.1-<r, a threshold chosen such that the entire set of 
Kepler observed stars could be searched within the fir st two years of the mi s sion with less than on e 
false positive transit detection owing to white noise (j Jenkins et al.ll2010bl : iBatalha et al.ll2010bl ). 
Threshold-crossing events are further screened and those passing a series of data validation steps 
are assessed by memb ers of the Keple r science team, includ ing individual visual inspection of each 
candidate light curve ()Wu et al.ll2010l : IBatalha et al.ll2010al ). Those that pass become planet can- 
didates promoted to Kepler Object of Interest status (KOI). The Kepler team's detection and 
validation process have proved to be remarkab ly effective, producing more than 1235 planet candi- 
dates orbiting 977 stars (jBorucki et al.ll2011bl ) (which would double the num ber of known planets), 



including 170 multi ple planet (multiplanet) systems (j Liss auer et al.ll201ll ). the first detection of 



Earth-sized planets (jFressin et al.ll2012l ; iGautier et alj|2012fl. and the detection of u nusual systems 



such as the discovery of circumbinary planets ( Welsh et al.l 2012 : Doyle et alJboill ) 



Only a few percent of these candidate discoveries have been confirmed to date via radial velocity 
measurements, tra nsit timing variations, or the statistica l rejection of false positives that mimic 
transit signals (e.g. iFressin et al.ll2012l : IGautier et al.ll2012l : iBorucki et al.ll2012l : iHowell et al.ll2012l : 
Bonomo et al.ll2012l ). The majority of the Kepler target stars are 12-16th magnitude, most too faint 
to enable efficient radial velocity follow-up of likely candidates using even the largest ground-based 
telescopes. The bulk of the KOIs will therefore likely remain unconfirmed. However, the Kepler 
planet candidate sample nonetheless allows a statistical assessment of the quality of the remaining 
candidates to be made, and for limits on the likely planetary population of the Milky Way to be 
derived. For example, a statistical analysis of Kepler candidates with periods of les s than 50 days 



aroun d Solar-type stars using a 10-<t signal detection threshold was performed by iHoward et al 



(|201ll ). The frequency with which stars have planets is found to increase with orbital period and 



decrease with radius as expected from core accretion models. The effective temperature of the star 
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was also found to be an important factor, with cool stars more likely to host planets. 



These results naturally depend on the performance of the Kepler transit detection and vet- 
ting processes. There remains a pressing need for an independent assessment of the ability of the 
combined automated TPS pipeline and data validation process to recover candidates. The Kepler 
light curves are complex, with many exhibiting significant structure which includes multiple oscil- 
lations as well as quasi-periodic oscillations and short-lived variations. Traditional transit searches 
and validation techniques looking for a periodic signal may potentially miss transit signals that 
are dominated by the natural variability of the star or the Kepler light curves may have artifacts 
or other systematics that might potentially compromise automatic fitting routines. We employ a 
novel method of searching for transit signals in the Kepler light curves via visual inspection. The 
human brain is a superb pattern recognition device and with minimal training can often outperform 
the most so phisticated machine learning devices and perform tasks not easily adapted to computer 
algor ithms (jSimpson et aDboid : lKhatibll201ll : bashti et allboiCll : iLintott et aDbood bood : iMendez 
2008). The detection of transits is essentially an exercise in pattern recognition, and is thus ideally 
suited to human classification. Transits that may be missed by traditional transit detection and 
vetting techniques may be identified by the human eye; highlighted by the serendipitous detection 
of a pr eviously unknown tr ansiting brown dwarf found by visual inspection of the Kepler light 
curves (Johnson et al. 2011 



While classifying light curves from 150,000 stars would be beyond any single classifier, by using 
the World Wide Web it is possible to gather multiple independent assessments of the same data set. 
This approach also allo ws for a careful ana lysis of the reliability of any individual classification. 
With Pla net Hunters^ dFischer et al] I2OI2I ) , part of the Zoonivers^l collection of citizen 



projects (ILintott et al 



2011 



science 



20081 ) . the Kepler light curves have been reviewed by eye for the 



signatures of extrasolar planets collectively by more than 173,000 volunteers to date with over 
24,000 citizen scientists participating in the analysis presented here. Although novel in the search 
for exoplanets, human classification and crowd-sourcing citizen science have been applied to a 
wide range of astronomic a l applications inc luding the morphological classi fication of galaxies (e.g. 



Lintott et al.ll201ll . 120081 : lLand et al.l 120081 ), the search for s upernovae (jSmith et al.1 1201 ll ). the 



i dentification of solar coronal mass ejections (jDavis et al]l2012h. cataloging of star forming regions 
([Simpson et al.ll2012l ). and cratering counts on the Moon (|Joy et al.ll201ll ) in the Zooniverse suite 
of proje cts, as well as for discovery of solar grains ent rapped in aero gel collected by the Stardust 
mission ([Mended l2008l ). and cratering counts on Mars (|Mendezl l2008). 



The success of the Planet Hunters has already been demonstrated bv lFischer et al.l (J2012J), who 
report, in a preliminary search of Kepler Quarter 1 light curves, the discovery and follow-up of two 
new planet candidates which were not identified as planet candidates in the Kepler team's analysis 



1 http://www.planethunters.org 
2 http://www.zooniverse.org 
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of the first four months of Kepler observations. Transits in the light curves of KIC 10905746 and 
KIC 6185331 were detected b y TPS but failed to pass the data verification stage and therefore 



not promoted to KOI status ([Fischer et al.l 12012 ). These discoveries, interesting in themselves 



illustrate the desirability of an independent check on the Kepler team's planet transit detection 
and validation scheme. 

This paper presents an independent assessment of the Kepler planet inventory, examining 
the abundance of > 2 R ffi planets on short period (< 15 days) orbits in the first 33.5 days of 
Kepler science observations (quarter 1), based on the classifications gathered from Planet Hunters. 
We detail how the classifications were obtained and processed and present our candidate detection 
scheme. We also assess the detection efficiency of the Planet Hunters project including a comparison 
to the Kepler inventory of planet candidates. 



2. Kepler Quarter 1 Data 



The Kepler Quarter 1 (Ql) dataset consists of observations for 156,097 stars spanning 2009 
May 13—2009 June 15. The data were obtained from the Kepler Mission Archival hosted by the 
Multi-Mission archive at STScI (MASlB). The majority of the light curves used in our analysis 
are from the 2010 June 15 data release. A subset of Ql targets, 5828 stars, had long cadence light 
curves publicly released after 2010 June. We obtained data for those stars from the 2011 February 
2 release. Du e to an error in uploadin g, five stars (KIC 8349808, 3656476, 6603624, 11446443 [KOI 
1.01 (Tres-2. loTkmovan et aPbood )]. KIC 11026764), comprising ~0.004% of the total Kepler 
sample, were not included in this analysis. 

Planet Hunters utilizes the long cadence observations from Kepler where measurements are 
effectively obtained once per 29.4-minute interval, presenti ng the scaled systema tic error-corrected 
flux outputted by the Kepler science processing pipeline ([Jenkins et al.l l2010bl ). The flux values 
and uncertainties used correspond to the Pre-Search Data Conditioning (PDC) PDCSAP_FLUX 
and PDCSAP_FLUX _ERR, previo usly labeled AP CORR FLU X and AP_COR_ERR columns 
of the released Kepler FITS tables (IFraquelli &: Thompson! l201ll ). The PDC correction removes 
systematic effects and long term trends from the light cu rves. For furth e r detai ls of the Kepler Ql 



observa tions and data processing we refer the reader to IBorucki et al.l (|2011al ) and iJenkins et al 
(hoiObllah. 



3 http: //archive, stsci.edu/kepler 
4 http: //archive, stsci.edu 
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3. Planet Hunters 

The Planet Hunters interface presents the volunteer with a single 33.5-day Ql light curve for 
a given star, plotting the star's relative brightness (Figure [1]). Error bars representing the 1-sigma 
measurement uncertainties are also displayed in the interface. The default setting is to show the 
entire 33.5 days at once. Volunteers have the ability to freely zoom the time axis or to scan the curve 
while zoomed in. A button allows the zoom on the y-axis (relative brightness) to automatically 
be adjusted to an appropriate level. The y-axis range spans from the minimum value to the mean 
level plus three times the standard deviation. The display is then adjusted to 5-85% of that range; 
this was found experimentally to produce a good result for the majority of curves. 

Visitors to the Planet Hunters site can begin classifying almost immediately. Basic training is 
provided via an inline interface, accessible with a single click from the Planet Hunters home page. A 
sample light curve with example simulated transits is shown before volunteers can review real light 
curves. More detailed training material is also available elsewhere on the site. It is possible to train 
and then to take part in the project without registering a Zooniverse account, although volunteers 
who are not logged in will receive repeated reminders to register. The site is designed so that 
volunteers are unable to select which curve to classify in order to minimize the risk of deliberate 
manipulation of input data. Light cur ves are priorit i zed ac cording to the stellar properties and 



previous classifications as described in iFischer et al.l (|2012l ). To ensure the volunteers have no 



prior knowledge about a given light curve that may effect their judgment, the KIC identifier is not 
revealed during classification; only an approximation of the star's stellar temperature, radius, and 
magnitude are presented to the reviewer. 

Classification proceeds via a decision tree, as detailed in Figure El The first series of questions 
produce a coarse categorization of the light curve's variability which is primarily intended to assist 
in identifying transits and in identifying any gaps or defects in the light curve that may have been 
introduced during the PDC process, typically due to removing a cosmic ray event (Jenkins et al 
2009). If the light curve has no visible gaps, volunteers are asked to categorize the star as 'variable' 
or 'quiet'. If the star is deemed variable, the volunteer is further prompted to characterize the 
variability as: 'regular' (showing periodic repeating pattern), 'pulsating' (rapid variations observed 
in star's brightness on the timescales of a few hours to days), or 'irregular' when no discernible 
pattern is seen. Representative light curves for each variability class are presented in Figure El 

Volunteers are then asked to mark any transit features visible in the light curve. Clicking a 
button produces a box on the light curve (Figure 1) positioned in the center of the review interface 
which can then be moved and resized. If necessary, more than one box can be added to each curve, 
and transits marked in error can be removed. The current zoom level of the interface determines 
the default box size. The box width and height initially correspond respectively to l/8th the x- 
axis range and half the y-axis range displayed. A second version of the interface, introduced on 
2011 Jan 31, allows in addition for transit boxes of any size to be drawn directly onto the curve, 
without the need to click a button first. In the upgraded interface, if a transit box already exists 
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when the button to add a new transit box is clicked, a box with the same dimensions as the one 
previously made is generated. To aid in refining the transit box dimensions, with a single click the 
improved interface will automatically center on a selected transit box, zooming in to a width twice 
the size of the region highlighted by the specific transit box. Due to the way that transit boxes 
are presented on the screen, only approximately eighteen boxes can be placed without hiding the 
button to submit the classification, effectively creating a limit on the number of transits that can 
be identified. This issue only significantly affects the marking of transits from planets with orbital 
periods less than ~2 days. 

Classifications are stored in a live Structured Query Language (SQL) database. For each entry 
question in the decision tree, the time stamp, user identification, light curve identifier, and response 
are recorded. If transit boxes are drawn, the position of the box center, width, and height of each 
box are recorded as well. Each of the 156,092 Ql light curves is examined and classified multiple 
times, by normally 10 and at least 5 classifiers including both logged-in and unregistered users 
before being retired and removed from circulation. The variability and gap questions are only 
asked of those volunteers registered with a Zooniverse account; non-logged in classifiers are asked 
only to identify transit-like features. Before the first classification and subsequently after every 
5 classifications, a non- logged classifier is reminded to create a Zooniverse account. If the user 
registers, subsequent classifications are attributed to the user in the classification database, but 
previously made classifications are labeled as from a non-logged in user. 

Once the classification is submitted, the Planet Hunters internal designation for the light curve 
is revealed to the user. Classifiers are then asked if they would like to discuss the particular light 
curve on the Planet Hunters Talk sit^l. Each Ql light curve has a dedicated page where users 
can write comments, add searchable Twitter-like hash tags, and link similar light curves together. 
This system is useful in identifying objects of particular interest and unusual light curves that 
are difficult to find via automatic classification schemes, a complement to the systematic analysis 
described here. 

110,997(71%) of the Kepler Ql light curves were completed with the original version of the 
interface and decision tree launched on 2010 December 16. We note that on 2011 Feb 1, the 
decision tree was modified for the inclusion of Kepler Quarter 2 data. All Quarter 2 light curves 
presented in the Planet Hunters interface have missing data gaps due to safe mode events or the 
Kepler spacecraft pointing towards Earth to transmit data. In the original interface, the light curve 
variability questions were skipped if the user had identified a gap in the light curve. The decision 
tree was modified such that if there was a gap in the light curve the variability questions would 
still be asked. In subsequent changes on 2011 March 9, the gap question was removed entirely. The 
responses to the gap question are not used in identifying possible planet candidates. 7,759 light 
curves received classifications after 2011 May 1 when the majority of the light curves presented 



http://talk.planethunters.org - The code is available under an open-source license at 
https: / /github.com / zooniverse/Talk 
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in the classification interface were Quarter 2 Kepler light curves. The appearance and quality of 
the Quarter 2 observations is different from that of Quarter 1 which may make Ql light curves 
identifiable to more experienced users. We address the introduction of Q2 light curves on Ql 
classifications and the impact of the interface modifications on the user responses and behavior in 
Appendix [XI 



3.1. Synthetic Transits 



In order to characterize our detection efficiency, simulated transit events were injected into 
a subset of the Kepler light curves and presented on the Planet Hunters classification interface. 
Synthetic transit light curves were created from the Ql light curves randomly sampling from the 
156,097 public 



Borucki et al 



Slawson et al 



y rele ased long cadence light curves excluding kn own Kepler planet candidates 
2011b), transit false p ositives ( Borucki et al. 2011bl ). and known eclipsing binaries 



2011 



Prsa et al. 



2011 



available online as of 2011 Aprij^. A total of 6494 synthetic 
light curves were created. Simulations were divided into 7 radii bins from 2-15 and 6 orbital 
period bins between 0.5 and 15 days. Tabled] details the number of synthetic light curves generated 
for each radii and period bin. 

Synthetic transit points are implanted into the Kepler light curve by subtracting t he a mount 
of flux decremented during the transit, following the equations of lMandel &; Agoll (|2002l ) and IWinn 
(|2010l ) for transit depth and duration and assuming no limb darkening. We neglect inclination and 
eccentricity effects assuming circular edge on orbits as viewed from Earth. We do not adjust the 
measurement flux errors for data points implanted with the transit event, providing a conservative 
and slight overestimate of the Poisson measurement noise. The starting time of the first injected 
transit is randomly chosen between the starting time of the light curve and one orbital period later. 
Each simulation has at least two injected transits present in the light curve. 

We adopt t he stellar rad i i and surface gravity (i.e. log(g)) values reported in the Kepler Input 



Catalog^ (KIC, brown et al" 



2011 



to compute the stellar mass and the transit depth. The KIC 
uses multi-band color photometry to distinguish between main sequence dwarf and cool giant stars 
in the Kepler field. The KIC performs this task reasonably well with only a small fraction of th e 
catalog estimated to be giants misidentified as dwarf stars (jBrown et alJl201ll ; iKoch et al.ll2010l ). 
The KIC provides a first estimate of the stellar properties of the Kepler target stars, with log(g) 
reliable to within 0.4 dex with a 0.25 dex rm s in comparison to high resolution spectra taken for 
a subs ample of KIC stars (jBrown et alj|201ll ). The reported uncertainty for the KIC stellar radii 



varies. 



Brown et al.l (|201ll ) note that subgiants misidentified as m ain sequences stars m ay have their 



radii underestimated by as much as a factor of 1.5-2. According to lHoward et al.l (|201ll ). stellar radii 



6 http: / /keplerebs. villanova.edu/ 

7 http: //archive. stsci.edu/Kepler/Kepler fov/search.php 



- 8 - 



are reliable with in an rms of 35%, w ith iBorucki et al.l (|2011bl ) quoting errors in stellar diameters 
as high as 25%. IVerner et al.l (|201ll ) find that the KIC systematically underestimates stellar radii 
by as much as 50% for stars with radii less than 1 R Q . Given this large uncertainty in the KIC 
values and differences in the reported error estimates, we assume the KIC is representative of the 
bulk properties of the Kepler stars and si mply employ the KIC calculated stellar parameters in the 



same manner as other previous works (e.g. iHoward et al. 



2011 



Borucki et al. 2011b: Latham et al 



2011 



)• 



The simulations are presented to logged-in users only and classified in the exact same manner as 
the real light curves. After the classification is submitted, the user is notified that the light curve was 
simulated with the injected transit points identified. We note that the simulations were classified 
on the Planet Hunters site after the completion of the majority of the Ql light curves and were 
presented alongside the Quarter 2 light curves in the second version of t he classification interface . 



Quarter 2 is substantially and notably different in data quality than Ql (jChristiansen et al.l 12011 



However, we expect these effects to be small and the complete simulation set to be a good test of 
volunteer behavior on the entire Ql data set. We discuss the impact of these effects and detection 
efficiency further in Appendix lAl 



4. Selecting Transit Candidates 

In this section we detail our algorithm for combining the individual classifications received for 
each light curve and the selection pipeline developed to produce a final catalog of transit candidates. 
Each light curve in our sample was examined and classified multiple times, with a minimum of 5 and 
a mean of 10 independent classifications. In total, 1,687,547 individual classifications were generated 
searching the Ql data and examining the synthetic light curves. 24,300 registered volunteers made 
Vo of those responses, with the remaining 2% from unregistered volunteers. Figure S] pl ots the 



distr i bution of user classificatio ns for registered users. As is common for online projects (jZachte 



20121 ; ICrowston Fagnotll2008l ). most volunteers typically classify only a few light curves, a median 
of 5 and mean of 68 light curves. 47% of logged-in users have classified fewer than 5 Ql light curves. 
Only 10% of classifiers have made more than 100 classifications. 



4.1. User Weighting 

Rather than combining results from all classifiers equally in a majority vote, the first step 
in converting the raw classifications to transit candidates is to determine a user weighting. The 
weighting scheme distinguishes those classifiers who are more sensitive to identifying transits and 
less prone to providing false positives in order to consider their responses more heavily than others 
when selecting transit candidates. We evaluate the ability of each classifier and assign a weight 
based on their tendency to agree with the majority opinion and for those who classified the synthetic 
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light curves, on their perf ormance identifying the simulated tr a nsit e vents. We follow a modified 



prescription developed bv lLintott et al.l (|2008l ) and lLand et al.l (|2008l ) for user weighting based on 



the visual morphological classifications of galaxies from the Galaxy Zoo project. 

We assign user weights in a two stage process. Initially, all users start out equal, with a weight 
of unity. We then use the results of the synthetic light curves to seed the initial weighting for those 
users that classified a synthetic light curve. Prom the synthetic transits we have a direct evaluation 
of an individual user's ability to identify transits for a given planet radii and orbital period. Only 
logged-in users are served synthetic light curves to classify. 6,307 (26% percent) of the registered Ql 
volunteers examined at least one synthetic light curve. A median of 2 and a mean of 10 simulations 
were classified by those 6307 volunteers. For every synthetic light curve and classifier who viewed 
the simulation, we evaluate how well the user identified the injected transits events. The synthetic 
transit is deemed marked if the middle of the injected event is within one of the user boxes drawn. 
A given user-drawn box only counts once. If the user draws a single box around two simulated 
transits, credit is only given for successfully identifying one transit. 

For each simulation a user classifies, his or her weight is incremented by the fraction of markable 
transits correctly identified. Typically the number of markable transits is the number of synthetic 
transits present in the light curve but for those few simulations with more than 18 transits (only 
14% of the total) where marking more would cause the submit button to disappear effectively 
limiting the transits that can be identified, the maximum number of markable transits is set to 
18, and the weight of any volunteer who correctly marked 18 or more transits is increased by one. 
If no synthetic transits are marked, the user weight is decreased by 0.2 times the fraction of the 
simulation's classifiers who marked at least one transit on the simulated light curve. User weights 
are normalized such that the maximum user weight attainable, if all synthetic transits seen were 
identified correctly, is 2. Thus the weighting scheme rewards more heavily users who mark correctly 
the majority of the injected transits and down-weights users more significantly for missing easily 
identifiable transits and less so for harder to perceive transit depths. 

At this point, user weights have only been modified for those logged-in users who have classified 
at least one synthetic light curve. All other registered volunteers still have an initial user weight of 
1. We next use an iterative scheme to assign weightings to all users based on the responses for all 
the real Ql light curves. We divide user responses into two classes: 'transit' or 'no transit' based 
only on whether or not a user drew a transit box. For a given Kepler star i and for each class j, a 
score Si(j), is then calculated. We define Si(j) as: 

Si(j) = y~]wfc /c=users who voted j for light curve i (1) 

* k 

where Wi is the sum of the users weights for all the users who classified light curve i. We then 
update Wk , the weight of user k based on the calculated light curve transit scores (sj(j)). A 
user whose behavior is in line with the majority weighted vote will be rewarded and up-weighted, 
whereas those that disagree with the majority opinion will be down-weighted. We adjust wt as 
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follows: 

Wk = -Tj- ^2 s «0 f° r light curve i chosen by user k) i= all light curves reviewed by user k 

(2) 

where is the number of Ql light curves classified by user k, and scaling factor A is chosen to be 
such that the mean user weight remains at 1. 

The light curve scoring and updating of the user weights is iterated until convergence is 
achieved, when the median absolute difference between the old and updated user weights is less 
than or equal to lxlO -4 . Classifications from non-logged in users are included in the weighting 
scheme to calibrate the other logged-in users but all non-logged in users are effectively treated in 
the weighting scheme as a single user with an unchanging weight of 1. All users have final weights 
greater than zero with 55% of the logged-in users having values greater than or equal to 1. The 
final distribution of user weights is plotted in FigureEl 



4.2. Candidate Selection 

Once a user weighting solution has been found, the Ql light curves are searched for transit 
candidates. In this analysis, our search focuses on the identification of planet candidates with radii 
greater than 2 R ffi and orbital periods less than 15 days. From this point forward, the synthetic 
light curves are treated in exactly the same as the real Kepler light curves. Table [2] summarizes 
the candidate selection process and the number of light curves and simulations remaining at each 
step. 



4-2.1. Transit Score 

We use the 'transit' scores, Si (transit), for each of the Kepler light curves as a measure of 
the light curve's likelihood to have a planet transit. Each synthetic light curve is scored using the 
previously calculated user weights to determine Si (transit). We plot the cumulative distribution of 
transit scores for the Ql Kepler light curves and simulations in Figure [6l 

We aim to include as many possible transit candidates without the number of false positives 
light curves overwhelming the list. A cut in Sj( transit) is made to create the initial list of potential 
transit candidates. This threshold was chosen after visual inspection of a r andom subset of the 



remaining light curves and light curves with transit-like events detailed in iFischer et al.1 (|2012l ) 
previously found during a preliminary search through the Ql data set. The science team found 
that at Sj( transit) = 0.5 the false positives were overwhelming the number of candidate transit 
detections. Therefore we select our potential planet candidates as those light curves with Sj( transit) 
> 0.5. 5,511 (3.5%) of the Kepler light curves have s«( transit) greater than 0.5. 
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4-2.2. Removing Single Transit Events 

We restrict our search to transiting planets with periods less than 15 days where there will 
be at least two transit events present in Ql. The 50% transit score threshold only discriminates 
between the likelihood of the light curve having a transit or not, and is not dependent on the 
number of transits present in the light curve. There will inevitability be light curves with single 
transit-like events due to longer period transiting planets (orbital periods greater than 15 days) 
and primary transits from eclipsing binaries with orbital periods greater than 15 days where the 
secondary eclipse will occur in future quarters of Kepler data. We remove the most obvious of these 
single transit-like events from our candidates list, discarding light curves where a single transit box 
was drawn by all classifiers who identified transits. Of the 5,508 candidates, 570 light curves were 
removed by this cut. These cut light curves were separatel y analyzed for poten tial transits and the 



first candidates from that analysis are further discussed in iLintott et al.l (|2012l ) 



4-2.3. Light Curve Variability 

Visual inspection of the remaining candidate light curves revealed that a significant portion 
were pulsators, light curves which exhibit rapid variations in the star's brightness on the timescales 
of a few hours to days. Many classifiers mistook the individual pulsations for transits. To further 
cull the candidates list of these false positives, we assess the light curve variability on timescales 
less than 33 days, placing a more stringent transit score threshold for the pulsating light curves. 

While classifying, users are asked to broadly characterize the observed variability of the Ql 
systematic error-corrected light curves as described in Section [3j We note this may not reflect a 
true assessment of the stars' intrinsic astrophysical variability. The Pre-Search Data Conditioning 
(PDC) applied to the raw Kepler light curves may remove real astrophysical signals from, and 
potentially inject high-frequency signals, into the light curves w hen correcting fo r instrumental 



system atics and optimizing the light curves for transit detection (jvan Cleave! l2010i : iJenkins et al 



2010b! ) . The Planet Hunters variability assessment serves as an evaluation of the bulk properties 



of the Ql light curve sample rather than the properties of individual Kepler target stars. 

We first group the light curves into quiet and variable classes, then further subdividing the 
variable stars into pulsating, regular, and irregular. We combine together the multiple user classi- 
fications by simply taking the majority, selecting the variability class with the highest fraction of 
votes. If the light curve has less than 3 classifications or a majority is not reached, we deem the 
light curve unclassified. Table [3] summarizes the variability properties of the Ql light curves based 
on Planet Hunters classifications. The majority of the Kepler light curves, 65.6% are quiet with 
the bulk of the variable stars being classified as pulsators (48.9% of variable light curves, 16.5% of 
the full sample). 



The majority answer does not necessarily mean the proper identification of the light curve's 
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variability. Little direct instruction is provided to the user when categorizing the light curve's vari- 
ability. The inline tutorial does not address the variability questions asked in decision tree, instead 
focusing on the identification of planet transits, but there are prototypes for each v ariability class in 



the he l p material tha t is ac cessibl e to the use r by a single click during classifying. iMcQuillan et al 



(|2012l ). ICiardi et al.l (|201ll ). and iBasri et al.l (1201(1) assess the p hotometric variabi lity of the Ql 
dataset to which we can compare. McQuillan et al.1 (120121 ) and IBasri et al.1 (120101 ) examine the 
raw pre-PDC processed light curves apply ing their own meth ods to remove systematic effects while 
preserving stellar variability signals, and ICiardi et al.l (|201ll ) examine the of variability the post- 
PDC processed light curves. Because users classify the PDC light curves, we c hoose to evaluate th e 
reliability of the Planet Hunters assessments for the Kepler Ql dataset with ICiardi et al.l (|201ll ). 
where we can make a direct comparison. 



Ciardi et al.l (120111 ) divide the variability from a sample of 126,092 dwarfs and 17,129 giants 



using a reduced chi-squared threshold of 2. Although ICiardi et al.l (|201ll ) group the light curves by 
spectral type and into dwarfs or giants, we repo rt the total fraction s for the entire sam ple. In Table 



1 11 we c ompare the variability fractions from the lCiardi et al.l (|201ll ) to this work. Like lCiardi et al 



(|201ll ) we only examine stars that have KIC entries for radius and temperature. We observe the 
same trend of light curve variability decreasing with increasing magnitude with 49.6% of stars with 
instrumental magnitudes less than 14 in our sample deemed variable. This is to be expected as 
measurement precision is highest for the brightest stars in the sample. 

The variability fractions of the light curves with transit scores greater than 0.5 are detailed 
in Table [5j Only 497, or 10%, of the candidate light curves had too few classifications to secure 
variability identifications. The pulsators are overrepresented in the candidates, composing 36% the 
sample, nearly double the amount in the entire sample as a whole, confirming the results from 
visual inspection of the candidates. It is not surprising that the largest source of false positives in 
our candidates list would be due to pulsators. With the user classifications producing an accurate 
picture of Ql light curve variability, we can use the Planet Hunters variability identifications to 
place a more stringent threshold for pulsators and remove pulsator light curves with transit scores 
less than 0.8 from the remaining list of planet candidates. Of the 4938 candidates, 1534 light curves 
were removed by this variability cut. 



4-2.4- Round 2 Review 



After the series of cuts described above, 340 4 candidate light cu r ves and 4974 sim ulations 
remain, including a mixture of eclipsin g binaries (jSlawson et al.ll201ll : iPrsa et al.l l201ll ). Kepler 
planet candidates (jBorucki et al.ll2011bl ). transit false positives, and potentially previously uniden- 
tified planet candidates. Planets with periods less than 15 days will have at least two transits during 
the Ql observations. In the analysis presented here, we are only interested in those light curves 
with two or more identified transits. The current selection criteria only determine whether or not 
transits are likely to present in a given light curve, not identifying the number of visible transits 
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found by the Planet Hunters classifiers. In order to sort through the remaining list of candidates 
and identify those light curves with at least two transits, a second classification interface was built 
and second round of review performed. 

In this simplified interface, the full 33.5-day light curve is plotted with the same horizontal and 
vertical range as presented in the main classification interface To assist the reviewer, the positions 
of all the user-drawn transit boxes are overlaid on the light curve in blue such that regions where 
there is more overlap between the transit boxes are shaded darker. Reviewers can toggle on and 
off the plotting of the transit box locations with a click of a button. The light curve is presented 
as a static image; Round 2 reviewers are not able to zoom in or adjust the resolution of the light 
curve like the main classification interface. Both real and simulated light curves are presented and 
reviewed blindly by the screeners. Unlike the main classification interface, simulated light curves 
are not identified after classification. 

Screeners for the Round 2 review were recruited from the Planet Hunters Talk site and through 
a blog post. These volunteers are asked if the presented light curve has two or more transits. The 
volunteer has three possible responses: 'y es ') ' no '> or 'maybe'. In Round 2, the reviewers are not 
asked to determine whether the transits are due to a single planet or to identify possible secondary 
eclipses within the light curve. Instead we instruct the user to ignore whether the transits have the 
same depth in the tutorial information and ask them to focus only on the number of transit-like 
events identifiable in the light curve. 

Round 2 review was performed in two stages; the first in September 2011 and the second 
in February 2012. A much smaller number of classifiers examined the light curves in Round 2 
review interface compared to the total number of individuals who reviewed Ql light curves and 
simulations in the main Planet Hunters interface. 137 volunteers participated in Round 2 review. 
Every candidate light curve and simulation received at least 5 and an average of 9 independent 
classifications. A total of 75,291 classifications were made in the Round 2 review interface with 
volunteers examining an average of 550 and median of 75 light curves and simulations. The ma- 
jority of Round 2 classifiers are experienced users, having classified a median number of 177 Ql 
classifications, with 55% of the Round 2 reviewers having classified more than 100 Ql light curves 
and simulations. Only 25 of the Round 2 reviewers had not previously classified a Ql light curve. 
72% of the classifiers have user weights greater than or equal to 1; therefore, we choose to simply 
tally the number of responses for each answer choice ('y es \ 'no', 'maybe'). If the candidate light 
curve received a majority of 'yes' votes, the light curve is passed on for further review and scrutiny 
by the Planet Hunters science team. 



4.3. Planet Candidates 



In total 1220 light curves, 0.78% of the Ql light curves, were passed on for further inspection . 
We note one light curve, KIC 8953426, which is a known eclipsing binary (jSlawson et al.ll201ll : 
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Prsa et al.ll201ll ). did not receive Round 2 review due to a processing error but was passed on with 
the other remaining candidates to inspection by th e Planet Hunters sc i ence team. At t his stage, 



we identify and remove reported eclipsing binaries (ISlawson et al 



with an eclipsing binary contaminating th e aperture listed in 



planet transit false positives identified bv lBorucki et al.1 ( 2011bl) and Fischer et al. ( 20121 ). Those 



2011 



Slawson et al 



Prsa et al.ll201ll ). sources 



(2011Q, and ad ditional 



remaining which are not listed as KOIs as of February 2011 (IBorucki et al.ll2011bl ) are visually 
inspected. 

The Planet Hunters science team reviewed the 75 remaining light curves, first verifying there 
were at least two transits present in the light curve. For those candidates with multiple transits 
visible, the Quarters 2-6 light curves were next inspected to confirm repeat transits in subsequent 
quarters. We note that Quarters 2-5 light curves are currently being analyzed by the Planet Hunters 
community. Odd/even transits were checked for depth to remove obvious unlisted eclipsing binaries. 
We also examined the Kepler target stars surrounding each of the remaining light curves for nearby 
eclipsing binaries that could be contaminating the photometric aperture and mimicking transit-like 
signals in the candidate light curve. If the transit times are correlated with that of a known eclipsing 
binary light curve in the vicinity or after examination of the individual pixel by pixel light curves 
for the star with PyKEj§ it can be seen that only a small subset of pixels exhibit the transit features, 
the candidate is deemed a false positive. Table [6] details the outcom e of the review. 23 sources, 
including a known RR Lyrae star (KIC 6186029, iBenko et al.ll2010i ). did not have two or more 
visible multiple transit-like events in the Ql light curve or were data processing glitches. 18 light 
curves did not have repeats in subsequent quarters of data, and 27 light curves have alternating 
transit depths indicating they are likely to be previously unknown eclipsing binaries or have transits 
correlated with an nearby eclipsing binary contaminating the photometric aperture. This left only 7 
surviving candidates (KIC 551108, 6616218, 8240797, 9729691, 10905746 11017901, 11551692) from 
our review not identified as KOIs in the Kepler analysis of Quarters 0-2 (jBorucki et al.ll2011bl ). All 
7 candidates have been subsequently identi fied in later runs of an updated version of TPS and data 
validation pipeline (jTenenbaum et al.ll2012l ) and promoted to KOI status by the Kepler team during 
a multi-quarter TPS analysis of Quarters 1-6 and data validation using products from Quarters 1-8 
(Batalha 2012- private communication,Batalha et al. 2012). Additionally KIC 5511081, 6616218, 
8240797, 9729691, 11551692 have been subsequently identified by the Kepler team as multiplanet 
systems with at least one > 1.9J2® planet candidate orbiting the host star in less than 15 days. 

All seven light curves are plotted in Figures [7] and In Figure [9] we provide a zoom-in of a 
chosen transit for each set of transits identified in the 7 light curves. Visually the science team 
could identify two separate sets of repeating transits in the mutli-planet KIC 8240797, 9729691, 
and 11551692 based on th e user drawn boxes. We note that KIC 10905746 is a short period planet 
candidate first reported bv lFischer et al.l (|2012l ). found in a pre liminary search of th e Planet Hunters 
Ql classifications. The second planet candidate reported by iFischer et al.l (J2012J) was not in this 



3 http://keplergo. arc. nasa.gov/PyKE.shtml 
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analysis since it has a period of 40 days with only a single transit in Ql. Additionally KIC 5511081 
was also ide ntified as a pl a net ca ndid ate by Planet Hunter s in an preliminary search of Q2 data as 
reported bv lLintott et all (fcoid ) and batalha et all (|2012l ). 



We do not s pecifically know the r easons why each of the 7 light curves failed to be promoted 
to KOI status in (IBorucki et alJl2011bl ). All 7 light cu rves were flagged as hav ing potential transits 
with the multi-quarter TPS analysis of Quarters 1-3 (jTenenbaum et al.ll2012l ). but this version of 
TPS was used to create the iBatalha et al.l ([20121 ) candidate list a nd was not implemented during 
the ([Borucki et alj|2011bl ) candi date selection. ThelBorucki et al.l (|2011bl ) version of TPS identified 
transits in individual quarters. iFischer et all (|2012l ) report that KIC 10905746 was identified by 
TPS but the automatic light curve fitting failed to converge to a solution and the star was rejected. 
4 of the 7 systems have two transiting planets with periods less than 15 days, this may have been 
an additional factor. Regardless, for the exact reason these candidates were not initially identified, 
they are now KOIs by th e multi-quarter TPS an d upgraded data validation scheme detailed in 
Batalha et all (120121 ) and iTenenbaum et all (J2012J) . The lack of any other additional candidates 
not currently KOIs highlights the effectiveness of the Kepler detection scheme and the increased 
detection efficiency from improvements to the Kepler detection pipelines. 



5. Detection Efficiency 

In this section, we discuss the detection efficiency of Planet Hunters for finding short period 
planets with radii larger than or equal to 2 R ffi and periods less than 15 days. We first assess 
the performance based on the detection of our sample of simulated transit light curves and then 
compare to the known sample of Kepler planet candidates. We explore the potential biases that 
may effect user responses and behavior and their impact on the calculated detection efficiency in 
Appendix [Al 



5.1. Synthetic efficiency 

Simulations were processed and screened by Planet Hunters users in a similar manner to the 
real Kepler light curves, and Table [2] summarize the simulation light curves that were retained at 
each stage in the planet candidate selection process. Those synthetic light curves that survived the 
transit score threshold, the single transit removal step, and the variability cut were shown in the 
Round 2 review. The simulations were shown with real light curves and at this stage users were 
not notified which light curves had injected transits or which ones were real Kepler light curves. 

To estimate the detection efficiency, we bin the simulations into the same radii and period 
bins used t o gene rate the light curves. Figure [10] (with Poissonian 68% errors as prescribed by 



Kraft et all Il99ll ) and Table [7] present the recovery rates for the 6494 generated synthetic light 



curves. We deem a simulation recovered if the synthetic light curve successfully passed Round 2 
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review stage. We find that greater than or equal to 4 R®, independent of orbital period, Planet 
Hunters is > 85% efficient at detecting short period planet transits. The detection efficiency begins 
to drop at 3-4 R® to 75% and for the 2-3 R® bin 40% of the synthetic light curves were recovered. 
For the larger planet radii bins in Figure [TU[ the majority of the missed simulations were around 
> 2 R Q stars where the transit depths are as low as those for < 3 R® planets around solar radii 
stars. For all the radii bins, the detection efficiency was relatively insensitive to orbital period. We 
note that we injected into a random sampling from the Ql light curves including giant stars, thus 
our synthetic recovery rate represents the Planet Hunters detection efficiency for the overall Kepler 
target star sample. 

We find that our detection efficiency is independent of the number of transits present in the 
light curve. Classifiers are are just as likely to identify and mark transits in a light curve if there 
are 2 or 18 transits visible. With our detection efficiency independent of orbital period, we can 
examine the recovery rate as a function of the relative transit depth for the synthetic light curves. 
The transit depth is the true observable in the light curve, depending on both the planet radius 
and radius of the host star. Figure [TT] presents the recovery rate of synthetic light curves as a 
function of the relative transit depth of the injected transits binned with a bin size of 3xl0 -4 for 
transits with depths less than 0.005. We find as expected, detection efficiency drops at small transit 
depths. Beyond a relative depth of 0.0013 (relative transit depth of a ~4 R® planet around 1R Q 
star), nearly all the simulated light curves are recovered. 3326 (97%) of the 3417 synthetic light 
curves generated with transit depths larger the 0.0013 pass the candidate selection process and 
Round 2 review. 



5.2. Comparison with the Kepler Planet Candidates 



A natural comparison to make is to the known list of KOIs identified by the Kepler team's 
detection scheme. Our candidate detection scheme was not tested or tuned to specifically identify 
known Kepler planet candidates. Thus we can use the population of known KOIs to serve as an 
additional check of the Planet Hunters detection sensitivity. We divide the KOIs into the same 
7 radii bins ranging in 2-15 R® and 6 orbital periods bins between 0.5 and 15 days as used to 
generate the synthetic light curves. Of the 1235 KOIs identified in the first two qua rters of Kepler 



observations, 408 planet candidates are associated with multi-transit systems by (IBorucki et al 



2011bl ). Very few of the multiplanet systems satisfy our criteria where we can be certain a single 
set of transits in Ql were identifiable to a Planet Hunters classifier, there fore we restrict our an alysis 
to only those currently identified to reside in single planet systems by iLissauer et al.1 (|201ll ) . 330 



KOIs fit our criteria and are used in our analysis presented here. We note that KOI 01.01 was 
missing from our Ql sample of light curves and was not examined by Planet Hunters classifiers. 

Figure [12] plots the Planet Hunters efficiency for the Kepler sample of short per i od pl anets. 
Error bars are taken as the Poissonian 68% uncertainty (as prescribed by Kraft et al.1 (Il99ll ) ) for 
the value in each radii/period bin. The number of planet candidates and the number recovered by 
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Planet Hunters is reported in Tabl e El The KOI was deemed detected if it passed the Round 2 



review process. 



Fischer et al.l (120121) had visua l ly insp ected the classifications from the 306 Kepler 



planet candidates announced by Borucki et al. ( 2010al ) finding that 2/3rds of the 1371 transits with 



planet radii between 1 and 10 R® were flagged by Planet Hunters users with only 10% of transit 
boxes marking spurious detections. From the past visual inspection and low spurious detection 
rate, we are therefore confident that at least one of the appropriate KOI transits in each of the 
Planet Hunters candidate KOI light curves was identified. We note that majority of our comparison 
sample of KOIs are lost due to the transit score threshold cut with Round 2 review eliminating 
only 21 light curves and the pulsator variability removing only 2 KOIs. 

Although the error bars are large, we find that above 4 R®, Planet Hunters is > 80% efficient 
at identifying transits. We also observe the same trend observed in the simulations. Less than 4 
R® the detection efficiency drops with a 20% reported efficiency for the 2-3 R® KOIs. Beyond 4 
R®, the KOI recovery rates are in agreement with the detection efficiency measured for the injected 
transits. Additionally we see the same result we reported for the simulated transits, the detection 
efficiency is relatively insensitive to the number of transits present in the light curve. The largest 
discrepancy occurs in the 2-3 R® bin, where only 20% of the Kepler planet candidates are recovered. 
40% of the simulated 2-3 R® light curves were identified by Planet Hunters classifiers. 



5.3. 2-3 R® Detection Efficiency 

To further explore this discrepancy between the KOI recovery rate and the synthetic detection 
efficiency in the 2-3 R® bin, we examine the detection efficiency as a function of relative transit 
depth for the 144 Kepler and 1799 synthetic 2-3 R® light curves. We expect that if the classifiers 
are able to detect a transit of a given depth from the Planet Hunters simulations, they should 
equally be able to detect a KOI of similar transit depth. The detection efficiency for the 2-3 R® 
simulated light curves as a function of relative transit depth is plotted as Figure [13 As expected 
if the Planet Hunters classifiers are marking transits and not spurious features in the light curves, 
detection efficiency decreases with decreasing transit depth. Higher efficiencies are seen for transits 
with larger depth for both the simulations and the KOI distribution. 

We test if there are any significant differences between the detection sensitivity for the synthetic 
distribution and that obtained from the Kepler planet candidates. We bin the 2-3 R® Kepler KOI 
and synthetic samples into transit depth bins of 1.5xl0 -4 in size. We generate 100,000 'synthetic 
KOI' distributions by randomly selecting from the larger synthetic light curve population, the same 
number of simulations as there are real 2-3 R® Kepler KOIs in each transit depth bin. For each 
'synthetic KOI' sample generated, we determine which of the synthetic light curves were correctly 
identified as having transits by Planet Hunters. We then total the number of simulations recovered 
by Planet Hunters in each bin for the sample and compare with a chi-squared test to the binned 
histogram of Planet Hunters recovered real 2-3 R® Kepler KOIs. We find there is no significant 
difference between the synthetic distribution and that obtained for the Kepler planet candidates. 
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Of the 100,000 'synthetic KOI' samples, 81.9% percent can be rejected as being drawn from the 
same population as the recovered KOIs at a significance greater than 68%, with only 9.6% rejected 
at the 95% confidence level. Therefore at the two-sigma level, the recovery rates for the Kepler 
KOIs and synthetic transits as a function of relative transit depth are consistent. 

Nonetheless, when we plot the distribution of detected simulations and Kepler candidates as a 
function of radius, we do see signi ficant differences. W e use the Kuiper variant of the Kolmogorov- 



Smirnov (KS) test, as detailed in iPress et al.l ([1992), to compare the 2-3 KOI transit depth 



distribution to that of the synthetic population. The significance of the computed D statistic is 
found by performing 10,000 realizations of the KS test by randomly selecting the number of real 
KOIs from the simulated transit depth distribution and comparing to the full simulated sample 
and computing the fraction of instances where the computed D statistic was higher than the D 
statistic measured for the real KOI distribution. We find the hypothesis that both synthetic and 
KOI distributions are drawn from the same distribution can be rejected at the 99.98% confidence 
level. As sensitivity to transit depth is the same in both cases, the discrepancy in the recovery 
rate at the 2-3 R® bin, can then be attributed to differences in relative transit depth distribution 
between the two samples. 

The depth of the transit is the squared ratio between the planet and stellar radii, and the 
simulations randomly sample the Kepler target stellar population and have a uniform distribution 
of planet radii between 2 and 3 R® bin. If there is a significant difference in the distribution 
of stellar radii and planet radii for th e 2-3 Rm KOIs t his would exp lain the difference in the 



distributions. A 2-dimensional KS test (jPress et al.lll992l : |Peacocklll983l ) of the real and simulated 



population's planet radii and stellar radii distributions rules out at greater than the 95% confidence 
level that the two populations are compatible. The significance of the D statistic is estimated in the 
same manner for the 1-dimensional case, drawing 144 planets and their associated host star radii 
randomly from the synthetic population and comparing to the parent synthetic population 10,000 
times and calculating the fraction of instances where the computed D statistic was higher than 
the D statistic measured for the real KOI distribution. Our simulation population fundamentally 
differs from the true underlying planet population, producing different transit depth distributions. 

The KOI planets are skewed towards smaller radii than the simulations. Additionally the 
KOI stellar radii are also slightly shifted towards lower radii compared to the simulations. These 
differences likely are representative of the true underlying planet population and formation environ- 
ments. Though we note that any selection effect in Kepler candidate selection, systematic errors 
in planet radii or stellar radii estimations would shift the planet radius distribution. 



6. Conclusions 



This paper presents results from our analysis of the first 33.5 days of Kepler science data, 
focusing on the detection via transits of planets larger than or equal to 2 R^ with orbital periods 
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shorter than 15 days. We use a set of 1,687,547 classifications provided by more than 24,000 citizen 
scientists using the Planet Hunters interface, we have performed a systematic search for short period 
planet candidates. After vett ing 75 potential candi dates identified via the Ql classifications that 
were not identified as KOIs in lBorucki et al.l (|2011bl ) with an examination of all currently available 
Kepler public data (Quarters 1-6), we find seven candidate light curves (KIC 551 108, 6616218, 



824079 7, 9729691, 10905746 11017901, 11551692) not listed as planet candidates bv lBorucki et al 



(|2011bl ) during the initial search of the first fo ur months of Kepler observations but were identified 



in search of the first 16 months of Kepler data (jBatalha et a 



2012 ) with an imp roved pipeline. KIC 



11017901 was previously discovered and characterized by ([Fischer et al.l 120121 ) and the remaining 
six additional candidates have been identified as containing at least one > 1.9 R^ planet candidate 
with transits repeating l e ss than 15 days in later runs of the Kepler pipeline searching Quarters 



1-6 (Batalha et al 



2012 



Lintott et al .1 120121 ) . All seven candidates require additional follow-up 



observations to confirm their planetary nature. No additional candidates not listed as a KOI were 
found. 

Using synthetic planet transits inserted into the data and the known Kepler sample of short 
period planet candidates, we have measured the detection efficiency of human classifiers to identify 
planet transits in the Kepler Ql light curves. Although performance drops rapidly for smaller radii, 
> 4 R.0 Planet Hunters is > 85% efficient at identifying transit signals for planets with periods less 
than 15 days for the Kepler sample of target stars. For 2-3 Rq planets, the recovery rate for < 15 
day orbits drops to 40%. Our high recovery rate of both >4 R^ simulations and KOIs and the lack 
of additional candidates not recovered by the improved Kepler TPS and data validation routines 
and procedures suggests the Kepler inventory of >4 R^ short period planets is nearly complete. 

Although our search was not specifically developed and optimized for identifying eclipsing 
binaries in the Ql Kepler sample, we can comment on the Kepler team's de tection efficiency for 



the catalog of d etached eclipsing binaries with orbital periods < 15 days (jSlawson et al 



Prsa et al. 



2011 



2011 



Our detection scheme is highly sensitive to finding detached eclipsing binaries 
where the separation distance between the two stars is much larger than their stellar radii, and 
the light curve has clearly defined and separated primary transits and secondary eclipses. Only in 
our final stage of candidate selection, after Round 2 review, are the transit depths considered; any 
potential new eclipsing binaries would be found during the science team review. Even for those 
light curves where the secondary eclipse is relatively obscured due to the vertical scaling needed to 
display the primary transit, the Planet Hunters volunteers would have marked the primary transit 
and in the final review stage the light would have been identified as an eclipsing binary. Most of the 
possible eclipsing binaries appear to be blends with a nearby eclipsing binary, leav ing only a few 



possib le new identifications. With 926 detached eclipsing binaries identified in the ISlawson et al 



(|201ll ) catalog as having periods less than 15 days, this would suggest the Kepler eclipsing binary 
catalog has identified almost all eclipsing binaries with periods less than 15 days. 



Planet Hunters is a novel and complementary technique to traditi onal planet transit m eth- 
ods. The previous discovery of new planet candidates highlighted in iLintott et al.l (|2012l ) and 
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Fischer et al.l (|2012l ) and the high detection efficiency for 4 R® and larger planets shows the ef- 
ficacy of visual review of light curves for detecting large transiting planets. With a detection 
efficiency greater than 85% for 4R^ and larger planets orbiting on periods less than 15 days, it is 
likely Planet Hunters can easily detect these large planets even for single transits present in the 
30-day light curve sections presented on the website. With the completion of Quarter 1, Quarters 
2-5, each spanning approximately 90 days, have been added to Planet Hunters once publicly re- 
leased, extending the observational baseline to ~400 days, and classification of these light curves 
is currently underway. With the public release of Quarters 1-6, Planet Hunters will be able to 
examine the abundance of single transit events that do not repeat and would not be identified by 
TPS as well as search for additional transit events that may have been missed from longer period 
planets with repeating transits in Q1-Q6. However, transits due to planets significantly smaller 
than 2 will likely remain difficult to detect; particularly Earth-sized planets will be nearly im- 
possible to identify by visual inspection with Planet Hunters due to the low signal to noise ratio 
of the individual transits. Improved user weighting schemes particularly promoting users who are 
more adept at identifying smaller radii planets may improve the detection efficiency at 2-3 R ffi in 
the future, but the majority of small rocky planet discoveries will likely remain in the realm of the 
traditional transit detection methods. Human review may have a niche and be particularly valuable 
for systems which are unusual, such as circumbinary planets, or difficult because the presence of 
Transit Timing Variations (TT Vs) prevents detection by automated rou tines which look for peri- 



odic repeats of transit features. iGarcia-Melendo fc Lopez-Moraled (|201ll ) highlight the difficulties 



such routines have with significant TTVs, and an application of the algorithm described here, which 
identifies individual transits rather than a periodic signal, to further releases of Kepler data may 
be able to catch them. 
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A. Appendix: Potential Biases in Candidate Selection and Detection Efficiency 

It is important to consider the potential biases in our analysis and determine how they may 
impact our candidate selection and user behavior. In this section, we discuss the effect of the 
interface upgrades and modifications on user classifications. Additionally we explore the potential 
biases that may effect the calculated synthetic detection efficiency 

The entirety of the simulations and a subset of the real Ql light curves, have been screened, 
since 2011 Feb 1, after the interface upgrade enabling volunteers to draw transit boxes directly 
on the displayed light curve. With classifiers now drawing their own transit boxes rather than 
being given a predefined box size, the frequency of boxes drawn and the typical transit box width 
and height likely changed. We do not place a stringent requirement on the number of transits 
identified or the transit width in the selection criteria to identify candidates. Thus, there should 
be little effect on our results. We only use the box width as a criteria during the weighting of the 
simulations, and since all the simulations were viewed in the updated interface there is no bias in 
the weighting scheme and the impact on candidate selection should be minimal. 

The subset of Ql light curves and the simulations reviewed after the completion of the majority 
of Ql data were interspersed with Q2 light curves which are scaled to a slightly different value and 
appear visibly different from the Ql light curves. In particular, the Q2 light curves contain gaps 
due to missing data where as Ql is continuous for a majority of the Kepler targets. This makes the 
Ql data and simulations possibly recognizable such that when a user views a light curve without 
a gap he or she assumes the light curve is a simulation and is being tested therefore behaving 
differently than when the same user classifies real Kepler light curves, i.e. the 'Hawthorne effect' 
(Mayo 1933; Adair, Sharpe &; Huynh 1989). For example, the classifier may mark more transit 



-22- 



boxes on features that the user is less certain about because he or she wants to correctly identify the 
'simulated' transits, and the simulations would then not reflect an accurate picture of the typical 
behavior of Planet Hunters volunteers. Most classifiers come to the Planet Hunters interface and 
classify a handful of light curves before leaving (see Figure HJ). Therefore most of the classifiers 
at any given time on the Planet Hunters classification interface will be unaware of there being a 
difference between the two sets of light curves and what that significance is. Their responses will 
be the same whether they are a classifying a synthetic light curve, Ql light curve, or Q2 light 
curve. Although a small subset of users who have examined many tens of light curves may have 
noticed the pattern, these users remain a small fraction of the entire Ql classifications. Only 13% 
of those users who classified a synthetic light curve classified more than 10 simulations. In addition 
the simulations and the remaining Ql light curves were both shown at the same rate. 1 in 5 light 
curves served was either a Ql or synthetic. Although the user might be able to identify the quarter 
of origin, they could not be certain if the light curve presented was a simulation or real data. We 
believe this effect is negligible for the majority of our synthetic light curve classifications, and that 
they reflect the typical behavior of Planet Hunters classifiers. 

We also note that simulations were added to the Planet Hunters interface after 2011 May. The 
simulations were classified by a different set of users than were classifying the bulk of Ql. 34% of 
synthetic classifiers never examined a Ql light curve. In order to estimate a detection efficiency, 
we make the assumption that behavior between the two sets of classifiers is the same. Although 
there may be some evolution of the classifier population as a subset continue to classify curves and 
improve their ability to identify transits, the nature of the site, is such there is always new users 
who are classifying light curves. On average a visitor spends only 18 minutes on the Planet Hunters 
site. These new classifiers would be just as likely to review a simulation and have the same level of 
expertise as the original set of Ql classifiers. Therefore, we feel they the synthetic classifiers serve 
as a representative sampling of the skill set and ability of the typical Planet Hunters classifiers, 154 
of the 330 single planet KOIs with periods less than 15 days and 2-15 were completed before 
the upgrade and were shown when the majority of the Ql data was present in the classification 
interface. We can use classifications as an independent test of our estimated synthetic detection 
efficiency. We find a similar recovery rate from these KOIs as for the injected transit light curves 
and full short period planet KOI sample. Therefore, we feel they the synthetic classifiers serve as a 
representative sampling of the skill set and ability of the typical Planet Hunters classifiers, and our 
detection efficiency calculated from the classifications of the synthetic light curves is appropriate 
for the entire Planet Hunters classifications. 
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Planet Radii Orbital Period (days) 

(fle) 0.5 < P < 2 2 < P < 4 4 < P < 6 6 < P < 8 8 < P < 10 10 < P < 15 



2 < R < 3 300 300 300 299 300 300 

3 < R < 4 199 200 200 199 200 200 

4 < R < 5 100 100 100 100 100 200 

5 < R < 6 100 100 100 100 100 198 

6 < R < 7 100 100 100 100 100 200 

7 < R < 10 100 100 100 100 100 200 
10 < R < 15 100 100 100 99 100 200 



Table 1: Distribution of synthetic light curves generated 
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Remaining Light Curves 




Real 


Simulations 


All 


156,092 


6494 


Transit Score Threshold 


5508 


4978 


Single Transit Removal 


4938 


4974 


Light Curve Variability Cut 


3404 


4974 


Round 2 Review 


1220 


4685 


Removing Known False Positives* 


379 




Removing Known KOIs 


75 




Science Team Inspection 


7 





Table 2: Remaining light curves at each stage of the planet candidate selection process for both the 
Ql Kepler light curves and generated simu lated light curves. * Fa l se positives included e clipsing bi- 



naries and other false pos i tives identified in lBorucki et al.l (|2011bl ). ISlawson et al.l (|201ll ) |Prsa et al 
(|201lh and lFischer et al.l (fcoid ). 



Variability 


# 


fraction of 


fraction of 


class 




full population 


variable population 


unclassified 


849 


0.005 





quiet 


102388 


0.656 





variable 


52855 


0.339 


1 


unclassified variable 


4260 


0.027 


0.081 


regular variable 


7683 


0.049 


0.145 


pulsating variable 


25820 


0.165 


0.489 


irregular variable 


15092 


0.098 


0.285 



Table 3: Break down of light curve variability in the Ql light curves based on Planet Hunters 
classifications 



<16 mag <14 mag 
Ciardi et al. f2011 '> 35.2% 60.8% 
This work 32.0% 49.6% 



Table 4: Variability frequencies from ICiardi et al.l (120111 ) compared to that derived by Planet 
Hunters classifications. We quote the fraction of variable light curves from this work for the subset 
of lig ht curves with viab le variability assessments and had temperatures and radii estimated in the 
KIC (brown et alJboilh 
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Variability 


# 


fraction of 


fraction of 


class 




c n i j_ ■ 

full population 


variable population 


unclassified 


111 


0.022 





quiet 


1835 


0.372 





variable 


2992 


0.606 


1 


unclassified variable 


386 


0.078 


0.129 


regular variable 


528 


0.107 


0.176 


pulsating variable 


1776 


0.360 


0.594 


irregular variable 


302 


0.061 


0.101 



Table 5: Break down of light curve variability in the Ql planet candidate light curves based on 
Planet Hunters classifications 



Single Event or Data Glitch or No Transits 

2161731 2557556 2569618 2992573 4261757 4846409 5437945 

6117719 6186029 6603956 7023575 7175826 7457184 8297860 8584610 

8719324 8911948 9364562 9589420 10428163 11282332 11913545 12455830 



Two or More Transit-like Features and No Consecutive Repeats in Q2+ Light Curves 
3337061 3937978 4552152 4739167 5212286 5561743 6032517 6219684 
7449421 7685675 7989590 8104436 8345153 8625821 9154469 9692612 
10467170 11516241 



Alternating Transit Depth Changes or Light Curve Contaminated by Eclipsing Binary 
2997178 3098197 3459199 3858879 3858917 4678875 5303346 5467124 
5471606 6032517 6045250 6182849 6543683 6612327 7877818 8095110 
8104030 9529744 9851142 10031907 10092312 10275880 10294613 10735575 
11200767 11607176 12691412 



Remaining Short Period Planet Candidates 



5511081 6616218 8240797 9729691 10905746 11017901 11551692 



Table 6: Outcome of Science Team Review- KIC 10905746 was previously identified bv lFischer et al 
(|2012h . KIC 5511081, 6616218, 8240797, 9729691, 10905746, 11017901,11551692 have been identi- 
fied in upgrades to the Kepler detection pipelines and are to be included in the next planet candidate 
release (Batalha 2012 personal communication, Batalha et al. 2012). KIC 5511081 was also iden- 
tified by Planet Hunters in a preliminary search of Q2 ob servations as repor ted by iLintott et al 
(|2012h . KIC 6186029 is a previously known RR Lyrae star benko et al.1 (|201ol ). 
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Planet Radii 

{Re) 






Orbital Period (days) 






0.5 < P < 2 


2 < P < 4 


4 < P < 6 


6 < P < 8 


8 < P < 10 


10 < P < 15 


2<R<3 


123/300 


120/300 


121/300 


129/299 


121/300 


115/300 


3<R<4 


153/199 


150/200 


148/200 


157/199 


136/200 


152/200 


4<R<5 


92/100 


85/100 


79/100 


88/100 


82/100 


165/200 


5<R<6 


95/100 


90/100 


78/100 


86/100 


84/100 


165/198 


6<R<7 


97/100 


99/100 


84/100 


87/100 


91/100 


174/200 


7 < R < 10 


94/100 


85/100 


88/100 


87/100 


93/100 


174/200 


10 < R < 15 


98/100 


88/100 


86/100 


88/99 


93/100 


176/200 



Table 7: Recovery frequency of simulated light curves 



Planet Radii 

(Re) 






Orbital Period (days) 






0.5 < P < 2 


2 < P < 4 


4 < P < 6 


6 < P < 8 


8 < P < 10 


10 < P < 15 


2<R<3 


1/7 


3/18 


8/30 


3/25 


4/21 


10/43 


3 < R < 4 


0/1 


5/6 


7/10 


3/5 


6/11 


8/14 


4 < R < 5 


1/2 


6/9 


1/3 


1/3 


2/2 


7/9 


5 < R < 6 


1/3 


2/3 


2/2 


3/3 


2/3 


2/3 


6 < R < 7 


2/2 


3/3 


4/5 


2/2 


1/1 


1/2 


7 < R < 10 


1/4 


13/14 


8/9 


4/5 


3/3 


5/6 


10 < R < 15 


5/7 


13/14 


7/7 


3/3 


2/3 


1/4 



Table 8: Recovery frequency of Kepler planet candidates (jBorucki et al.l l2011bl ) with periods 
between 0.5 and 15 days and radii between 2 and 15 R® 
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Front page (top) and main classification interface (below) of the Planet Hunters website. 
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Fig. 2. — The decision tree that a Planet Hunters classifier is presented with when reviewing a 
light curve. 
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Fig. 3. — Examples light curves in each class. Reported error bars are overplotted in gray. For 
each light curve presented 75% or greater of the classifications selected the specified type. 



-32- 



CD 
on 

ZD 



0.30- 



0.25- 



<D 

c 
X 

% 0.2C 
c 
_TT3 

"S 0.15 



T 1 I - 



n i n 



n i r 



n i n 



n i r 




40 60 

Number of Classifications 



80 



100 



Fig. 4. — Distribution of classifications used in this analysis per registered Planet Hunters user 
with a bin size of 2. The plotted distribution is truncated at 100 classifications for resolution. Only 
10% of all registered volunteers make more than 100 classifications. 
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Fig. 5. — Distribution of Ql Planet Hunters user weights binned in 0.5 bins. 



- 34 - 




s(transit) 




s.(transit) 



Fig. 6. — Cumulative distribution of transit scores for the Ql Kepler light curves (top) and 
simulations (bottom). 
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Fig. 7. — Short period candidate light curves remaining after Round 2 Review and visual inspection. 
KIC 10905746 previously identified in Fischer et al 2012 and the remaining three candidates. All 
light curves were ide ntified in later versions of the Kepler pipeline as KOIs in the latest KOI release 
(jBatalha et al.ll2012l ). KIC 11551692 and 6616218 have been subsequently identified by the Kepler 
team as multiplanet systems with at least one > 1.9R<^ planet candidate orbiting the host star in 
less than 15 days (Batalha 2012 -personal communication, Batalha et al. 2012). The light curves 
have been normalized and a linear trend has been removed. 
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Fig. 8. — Short period candidate light curves remaining after Round 2 Review and visual inspection 
continued. All presented light curves were subsequently identified by the Kepler team as multiplanet 
systems with at leas t one > 1.9.R® planet candidate orbiting the host star in less than 15 days 
(jBatalha et al.ll2012l ). KIC 55110 81 was also id e ntified by Planet Hunters in a preliminary search 
of Q2 observations as reported bv lLintott et al.l (|2012l ). 
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Fig. 9. — Zoom-in of selected transits for each set of transit identified visible in short period candi- 
date light curves remaining after Round 2 review and visual inspection. Visually the science team 
could identify two separate sets of repeating transits in the mutli-planet KIC 8240797, 9729691, 
and 11551692 based on the user drawn boxes We note that the snapshot of KIC 8240797 contains 
two independent transit events. 
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Fig. 10. — Efficiency recovery rate for simulated planet transits with orbital periods between 0.5 
and 15 days and radii betwee n 2 and 15 R$. Error bars are taken as the Poissonian 68% uncertainty, 
as prescribed bv lKraft et al.l (|199ll ). for the value in each radii/period bin. 
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Fig. 11. — Recovery rate for simulated planet light curves for transits with depths less than 0.005, 
with orbital periods between 0.5 and 15 days and radii between 2 and 15 R®, as a function of 
relative transit depth binned with a bin size of 3x10 . Error bars a re tak en as the Poissonian 68% 
uncertainty for the value in each bin, as prescribed bv lKraft et al.l (|199ll ). 
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Fig. 12. — Recovery frequency of Kepler planet candidates (jBorucki et al.l l2011bl ) with orbital 
periods between 0.5 and 15 days and radii between 2 and 1 5 R® Error bars are taken as the 
Poissonian 68% uncertainty (as prescribed bv lKraft et aJj ()199ll ) ) for the value in each radii/period 
bin. 
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Fig. 13. — Recovery efficiency as a function of relative depth for 2-3 simulations. For resolution 
th e plot is truncated at .002. Error bars are taken as the Poissonian 68% uncertainty, as prescribed 
by lKraft et al.l ()199ll ). for the value in each bin. 



